Purpose

Phase 1 of #2709 — extends ModelOpt FP8 support to video-gen models. #2913 covers Phase 1 for image-gen (Flux, Flux2-Klein, Qwen-Image, HunyuanImage-3); this PR adds the video-gen counterpart for both HunyuanVideo-1.5 and Wan2.2 TI2V-5B using the same loader infrastructure.

Builds on:

• （Phase 1）Add ModelOpt FP8 auto-detect support for diffusion checkpoints #2709 #2913 (Phase 1, image-gen) — ModelOpt FP8 checkpoint auto-detect + diffusers loader adapter (cherry-picked into this branch; rebases away when （Phase 1）Add ModelOpt FP8 auto-detect support for diffusion checkpoints #2709 #2913 merges)
• [Quant] Wire quant_config through HunyuanVideo-1.5 and Wan2.2 DiT for online FP8 #2920 (referenced, will not merge) — DiT quant_config wiring for HV-1.5 + Wan2.2 (extracted into this PR; [Quant] Wire quant_config through HunyuanVideo-1.5 and Wan2.2 DiT for online FP8 #2920 stays as online-FP8 ablation reference)

Changes

DiT wiring (extracted from #2920)

• hunyuan_video_15_transformer.py + pipelines — HunyuanVideo15Attention, HunyuanVideo15TransformerBlock, HunyuanVideo15Transformer3DModel accept quant_config / prefix; threaded to to_qkv, to_out[0], add_kv_proj, to_add_out, ff, ff_context.
• wan2_2_transformer.py + wan2_2_vace_transformer.py + 4 pipelines — WanSelfAttention, WanCrossAttention, WanFeedForward (+ ColumnParallelGELU), WanTransformerBlock, WanTransformer3DModel, VACE variant. Factories (create_transformer_from_config, create_vace_transformer_from_config) accept optional quant_config.
• Modulation (raw nn.Linear / scale_shift_table), patch embedders (Conv3d), time/text/image embedders, proj_out, and the HV-1.5 token refiner stay full precision.
• The aggressive skip patterns from [Quant] Wire quant_config through HunyuanVideo-1.5 and Wan2.2 DiT for online FP8 #2920 (attn1/attn2 quant_config=None on Wan2.2) are not applied here — that was an online-FP8 workaround; static calibration handles it.

ModelOpt FP8 helpers

• examples/quantization/quantize_hunyuanvideo_15_modelopt_fp8.py — HV-1.5 calibrator. Force-exports FP8 weights, patches quant_algo: FP8, hides quantizers during save. MHA quantizers off by default.
• examples/quantization/quantize_wan2_2_modelopt_fp8.py — Wan2.2 TI2V-5B calibrator. Same design.
• examples/quantization/check_modelopt_fp8_export.py — verifier. Reads safetensors header dtypes, checks quant_algo: FP8, classifies scale granularity (per-tensor / per-channel / per-block).
• vllm_omni/model_executor/stage_configs/hunyuan_video_15_dit_fp8.yaml + wan2_2_ti2v_dit_fp8.yaml — serving stage configs with auto-detect.

Adapter (this PR also fixes a general-purpose bug in #2913's adapter):

• modelopt_fp8.py:_get_weights_mapper now walks submodules to aggregate hf_to_vllm_mapper from whichever sub-module defines it. The adapter is instantiated with the whole Pipeline, so model-specific remaps (like Wan2.2's ffn.net.0. → ffn.net_0.) must be discovered on the transformer submodule, not the top-level Pipeline. Fixes silent-noise output that occurred on Wan2.2 ModelOpt FP8 before this change.
• WanTransformer3DModel.hf_to_vllm_mapper added with that remap.

Both calibrators share --weight-block-size 'M,N' for block-wise FP8, and the same fallback pattern: _force_export_quantized_weights + _patch_quant_config + hide_quantizers_from_state_dict — because ModelOpt's export_hf_checkpoint doesn't handle diffusers-video checkpoints natively.

Validation — HunyuanVideo-1.5 (1×H100 80GB, T2V 480×832, 33 frames, 30 steps, seed=42)

torch.compile enabled (default).

Engine signals confirming the path is wired correctly:

• factory.py: Building quantization config: fp8 → Building quantization config: modelopt — auto-detect upgraded the user's --quantization fp8 flag to ModelOpt based on quant_algo: FP8 in transformer/config.json
• data.py: Auto-detected quantization 'modelopt' from model config
• __init__.py: Selected CutlassFP8ScaledMMLinearKernel for ModelOptFp8LinearMethod — the ModelOpt FP8 kernel selected

Visual comparison — HunyuanVideo-1.5

BF16 baseline:

ModelOpt FP8 (this PR):

Same prompt ("A dog running across a field of golden wheat."), same seed, same sampling params. Output is BF16-equivalent — no detail collapse or composition drift like the online FP8 path showed in #2920.

Validation — Wan2.2 TI2V-5B (1×H100 80GB, T2V 704×1280, 49 frames, 30 steps, seed=42)

torch.compile enabled (default).

Engine signals:

• factory.py: Building quantization config: fp8 → modelopt (auto-detect fired)
• data.py: Auto-detected quantization 'modelopt' from model config
• __init__.py: Selected CutlassFP8ScaledMMLinearKernel for ModelOptFp8LinearMethod
• Zero unloaded weight_scale warnings after the hf_to_vllm_mapper fix for Wan2.2's ffn.net.0. → ffn.net_0. diffusers↔vllm-omni name remap.

Visual comparison — Wan2.2 TI2V-5B

BF16 baseline:

ModelOpt FP8 (this PR):

Same prompt ("A dog running across a field of golden wheat."), same seed, same sampling params. Output is BF16-equivalent.

How to use

Pre-calibrated checkpoints are published on HF Hub so reviewers can test without recalibrating:

• shunyang90/HunyuanVideo-1.5-480p-ModelOpt-FP8
• shunyang90/Wan2.2-TI2V-5B-ModelOpt-FP8

Option A: use the published checkpoints (no calibration needed)

# HunyuanVideo-1.5
python examples/offline_inference/text_to_video/text_to_video.py \
    --model shunyang90/HunyuanVideo-1.5-480p-ModelOpt-FP8 \
    --quantization fp8 \
    --prompt "A dog running across a field of golden wheat." \
    --height 480 --width 832 --num-frames 33 \
    --num-inference-steps 30 --seed 42 --guidance-scale 6.0 \
    --output outputs/hv15_modelopt_fp8.mp4

# Wan2.2 TI2V-5B
python examples/offline_inference/text_to_video/text_to_video.py \
    --model shunyang90/Wan2.2-TI2V-5B-ModelOpt-FP8 \
    --quantization fp8 \
    --prompt "A dog running across a field of golden wheat." \
    --height 704 --width 1280 --num-frames 49 \
    --num-inference-steps 30 --seed 42 --guidance-scale 5.0 \
    --output outputs/wan22_modelopt_fp8.mp4

Option B: calibrate from BF16 yourself (reproducibility / custom prompts)

# 1. Install
pip install 'nvidia-modelopt[all]'

# 2a. Calibrate HV-1.5 (~10–15 min on 1×H100)
python examples/quantization/quantize_hunyuanvideo_15_modelopt_fp8.py \
    --model hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v \
    --output ./hv15-480p-modelopt-fp8 --overwrite

# 2b. Calibrate Wan2.2 TI2V-5B (~10 min on 1×H100)
python examples/quantization/quantize_wan2_2_modelopt_fp8.py \
    --model Wan-AI/Wan2.2-TI2V-5B-Diffusers \
    --output ./wan22-ti2v-modelopt-fp8 --overwrite

# 3. (optional) Verify
python examples/quantization/check_modelopt_fp8_export.py --output ./hv15-480p-modelopt-fp8
python examples/quantization/check_modelopt_fp8_export.py --output ./wan22-ti2v-modelopt-fp8

# 4. Serve — auto-detect upgrades --quantization fp8 to ModelOpt FP8
# (same invocation as Option A, just pass the local output path as --model)

Test Plan

HunyuanVideo-1.5

• Calibration script completes on 1×H100 — 648 weights converted to FP8
• Checker reports quant_algo: FP8, 648 F8_E4M3 tensors, per-tensor scale granularity
• On-disk transformer 10.45 GiB (−66% vs 31.02 GiB BF16)
• Loads via （Phase 1）Add ModelOpt FP8 auto-detect support for diffusion checkpoints #2709 #2913's adapter (Auto-detected quantization 'modelopt')
• End-to-end inference produces valid video; visual parity with BF16
• Memory −15%, wall-clock −14% vs BF16

Wan2.2 TI2V-5B

• Calibration script completes on 1×H100 — 300 weights converted to FP8
• Checker reports quant_algo: FP8, 300 F8_E4M3 tensors, per-tensor scale granularity
• Loads cleanly via （Phase 1）Add ModelOpt FP8 auto-detect support for diffusion checkpoints #2709 #2913's adapter (after the hf_to_vllm_mapper fix — see adapter change below)
• End-to-end inference produces valid video; visual parity with BF16
• Memory −21%, wall-clock −16% vs BF16
• On-disk transformer 4.76 GiB (−77% vs 21.22 GiB BF16)

Both

• Pre-commit (ruff, format, typos) — passing
• torch.compile enabled (default) on both BF16 and FP8 for fair comparison
• HV-1.5 I2V variant + Wan2.2 I2V / T2V-A14B / VACE — wiring threaded, calibration untested

Known limitations

• Per-block static FP8 is calibrated-correct but not serveable yet. Upstream vLLM's ModelOptFp8Config / ModelOptFp8LinearMethod only dispatches per-tensor scales — a block-wise checkpoint crashes at load with a shape-mismatch assertion in parameter.py:_assert_and_load. Per-tensor serving is the shippable path; --weight-block-size is kept in the calibrator for when upstream gains block-wise dispatch.
• HV-1.5 and Wan2.2 aren't in ModelOpt's recognized-model registry — QKV fusion is skipped and we hand-roll the weight-export path. Works, but means fewer of ModelOpt's standard diffusion optimizations.
• MHA quantizers (K/V/softmax) off by default — attention numerics on long video sequences were sensitive even with static scales (empirically in [Quant] Wire quant_config through HunyuanVideo-1.5 and Wan2.2 DiT for online FP8 #2920 ablation).

Follow-ups (still Phase 1, other video/variant coverage)

• Wan2.2 T2V-A14B / I2V-A14B MoE variants (need 2×H100)
• Wan2.2 VACE variant (wiring threaded; calibration helper needs VACE-specific prompts)
• HunyuanVideo-1.5 720p + I2V variants
• Block-wise static FP8 serving once upstream vLLM dispatches on strategy: block
• Publish calibrated checkpoints to HF Hub under vllm-project-org/

Depends on #2913. References #2920 (online-FP8 ablation reference, will not merge).

cc @baonudesifeizhai @hsliuustc0106 @ArtificialRay